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1 .  Introduction 


1.1.  Background 

The  US  Air  Force  Armstrong  Laboratory  at  Brooks  Air  Force  Base  in  San  Antonio, 
Texas,  is  researching  the  application  of  the  Air  Force  artificial  intelligence  technology  in 
public  schools  through  a  series  of  computer-based  tutors  using  Intelligent  Tutoring 
Systems  (ITS)  technology 

Wright  Laboratory  at  Wright-Patterson  Air  Force  Base  signed  a  Memorandum  of 
Understanding  with  all  of  the  Air  Force  superlabs  to  support  Armstrong  Laboratory's 
research.  Wright  Laboratory  then  enlisted  the  assistance  of  the  Alliance  for  Education 
and  awarded  the  Alliance  a  grant  to  administer  and  implement  the  local  component  of 
the  project. 

The  Alliance  for  Education  is  a  nonprofit  organization  which  is  a  coalition  of  industry, 
education  and  government,  acting  as  a  third-party  advocate  to  improve  education.  The 
Alliance  for  Education  is  independent  of  local  school  districts  but  works  closely  with 
them.  The  Alliance  for  Education  developed  the  title  "Project  F.A.S.T.  Track" 
(Fundamental  Academic  Skills  Training)  for  reference  to  the  local  component  for  this 
national  project. 

Wright  Laboratory  originally  awarded  the  grant  to  the  Alliance  for  Education  in  1992  for 
one  year  with  a  renewal  option  for  two  additional  years.  Delays  in  software  development 
at  Armstrong  Laboratory  resulted  in  an  extension  of  the  project.  This  report  covers  the 
activities  of  Year  Four  and  Five  and  reprises  major  events  in  each  of  the  three  previous 
years  of  the  project.  Refer  to  WL-TR-94-4023  for  information  concerning  Year  One  of 
the  project,  WL-TR-95-4005  for  information  concerning  Year  Two  and  WL-TR-97-3808 
for  information  concerning  Year  Three. 

Activities  for  the  first  year  involved  site  selection,  site  preparation,  teacher  training  and 
support,  provision  of  technical  support,  public  relations  and  program  evaluation. 
Activities  for  Years  Two  and  Three  involved  research  on  the  Algebra  word  problem 
solving  tutor  and  an  English  reading  and  writing  tutor,  training  additional  teachers, 
retraining  past  teachers  on  operation  of  new  versions  of  the  software,  installation  of 
additional  equipment,  provisions  for  technical  support,  public  relations,  program 
evaluation  and  preparations  for  the  Life  Science  tutor. 
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1.2.  Definition  of  Terms 


Artificial  Intelligence  (AI)  Technology:  Computer  programs  that  attempt  to  achieve 
some  type  of  intelligent  behavior. 

Intelligent  Tutoring  Systems  (ITS)  Technology:  Application  of  artificial  intelligence  that 
enhances  the  power  of  computer-based  instruction  by  acting  like  an  expert  private  tutor. 

ISIS  (Instruction  in  Scientific  Inquiry  Skills):  Life  science  tutor  and  the  focus  for  Year 
Four  research. 

Maestro:  Writing  process  tutor  that  is  the  revision  of  R-WISE  and  the  focus  for  Year 
Five  research. 

R-WISE  (Reading  and  Writing  in  a  Supportive  Environment):  Reading/writing  tutor 
used  through  Fall  1995;  revised  to  become  MAESTRO. 

WPS  (Word  Problem  Solving):  Pre-algebra  tutor  and  the  focus  of  Year  One  research. 
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2.  Local  Project  Objectives 


I.  To  obtain  research  data  on  the  effectiveness  of  the  Intelligent  Tutoring 

System  (ITS)  of  the  Fundamental  Skills  Training  Program  for  Armstrong 

Laboratory  by  establishing  regional  testing  sites 

•  Primary  responsibility  for  the  research  rests  with  the  Air  Force  Armstrong 
Laboratory  in  San  Antonio,  Texas.  Wright  Laboratory  at  Wright-Patterson 
Air  Force  Base  signed  a  Memorandum  of  Understanding  with  all  of  the  Air 
Force  superlabs  to  support  Armstrong  Laboratory's  research.  Wright 
Laboratory  then  enlisted  the  assistance  of  the  Alliance  for  Education  and 
awarded  the  Alliance  a  grant  to  administer  and  implement  the  local 
component  of  the  project. 

•  F.A.S.T.  Track  computer  laboratories  have  been  established  and  have  been 
functional  at  Dayton  Dunbar  and  Trotwood-Madison  High  Schools  since 
1992. 

•  The  project  collected  data  from  both  Dayton  Dunbar  and  Trotwood-Madison 
High  Schools  and  forwarded  it  to  Armstrong  Laboratory.  It  included 

=>  English  and  mathematics  teachers’  pre-  and  post-attitude  surveys. 

=>  English  students'  on-line  pre-and  post-tests. 

=>  Mathematics  students'  on-line  pre-and  post-tests. 

=>  Science  students'  on-line  pre-and  post-tests. 

=>  Mathematics,  science  and  English  students'  on-line  journals  and 

"thought  logs"  in  which  they  recorded  personal  observations  regarding 
lab  activities  and  the  tutors. 

•  In  cooperation  with  Armstrong  Laboratory,  the  local  project  research  team 
conducted  qualitative  research  involving  science  teachers  and  students. 
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II.  To  deliver  an  individualized  instruction  through  transferring  the  technology 

of  artificial  intelligence  applications  to  two  public  education  systems  in  the 

Dayton  area  in 

(a)  a  pre-algebra,  word  problem-solving  tutor 

(b)  an  English  reading  and  writing  tutor 

(c)  a  life  science  tutor 

•  Each  school  lab  contains  28  networked  computer  stations  for  students  to  use. 

•  In  fall  1 996  Armstrong  Laboratory  provided  the  Alliance  with  versions  of 
ISIS  and  Maestro  tutors  for  use  in  the  Dunbar  and  Trotwood-Madison  High 
School  F.A.S.T.  Track  computer  labs.  The  tested  and  revised  WPS  tutor  was 
also  available  for  student  use. 

•  English  teachers  worked  with  students  in  preparation  for  their  Maestro 
laboratory  experiences.  The  skills  were  applied  in  four  sequential  areas  of 
writing 


=>  Prewriting  focuses  on  goal  setting  and  planning.  Tools  include  cubing 
(considering  a  topic  from  different  perspectives),  note  taking, 
clustering  (brainstorming  ideas  and  grouping  them  logically)  and 
outlining. 

=>  Drafting  helps  students  take  their  organized  ideas  and  compose 
sentences.  This  segment  of  the  tutor  provides  for  individual  student 
differences  by  selecting  one  of  three  modes-  -guided,  supported  or 
independent. 

=>  Editing  stresses  both  revision  and  publication  of  students'  work. 

•  Science  teachers  selected  curriculum  parallel  modules  to  increase  scientific 
inquiry  skills  as  they  pertain  to  ecological  topics.  Working  through  a 
personally  designed,  matrix  students  were  challenged  to 

=>  State  a  testable  hypothesis 

=>  Design  an  experiment  to  test  the  hypothesis 

=>  Conduct  the  experiment  simulated  in  the  ecosystem 

=>  State  a  conclusion  about  the  experiment 

=>  Accept  or  reject  the  hypothesis 
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DDL  To  support  school  districts'  efforts  to  increase  student  test  scores  on  the  Ohio 

Proficiency  Test  in  mathematics,  read  and  writing,  and  science. 

•  During  Year  Five  local  evaluators  focused  on  two  issues  relating  to  this 
objective: 

=>  Correlation  between  Ohio  Proficiency  Test  performance  and  the  tutor 
usage; 

=>  Relationship  between  student  performance,  environment,  technologies 
(both  hardware  and  software)  and  student  behavior. 

•  Trotwood-Madison  High  School  teachers  continued  to  use  the  WPS  lab 
experience  to  support  students  who  have  not  passed  the  mathematics  portion 
of  the  OPT  (Ohio  Proficiency  Test). 
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3.  Alliance  Responsibilities 


I.  ENSURE  TRAINING  AND  SUPPORT  FOR  TEACHERS  AND  SITE 
COORDINATORS 

•  Held  review  and  follow-up  sessions  with  Trotwood-Madison  and  Dayton 
biology  teachers,  site  coordinators,  and  a  Dayton  school  district  science 
specialist  to  supplement  the  ISIS  training  sessions  held  in  San  Antonio. 

•  Regularly  communicated  with  coordinators  and  met  periodically  with  teachers 
and  district  personnel  to  ensure  adequate  support.  Forwarded  information  to 
Armstrong  Lab. 

•  Reimbursed  Dayton  and  Trotwood-Madison  districts  for  the  equivalent  of  one 
class  period  per  day  of  each  site  coordinator's  time  to  ensure  adequate  time  to 
perform  duties. 

•  Paid  stipends  to  participating  English  and  biology  teachers  and  site 
coordinators  in  recognition  of  their  additional  responsibilities. 


II.  FACILITATE  THE  LOCAL  PROJECT  TEAM  TO  IMPLEMENT  AND 

OVERSEE  THE  PROJECT  GOALS  AND  ENSURE  ADEQUATE  STAFF 

AND  TECHNICAL  SUPPORT. 

•  Served  as  liaison  for  teachers,  district  administrators,  and  Air  Force  personnel 
at  both  Wright  and  Armstrong  Laboratories  to  ensure  research  issues  were 
addressed. 

•  Contracted  with  the  University  of  Dayton  Research  Institute  for  the  services 
of  Katie  Thorp,  associate  research  engineer,  to  provide  technical  assistance  to 
the  project  and  to  assist  the  local  research  team. 

•  Contracted  with  Select  Tech  Services,  Inc.  to  ensure  that  qualified  computer 
technicians  were  in  the  computer  laboratories  with  teachers  and  students  at  all 
times  for  immediate  resolution  of  any  hardware  or  software  problems  or 
questions  and  to  protect  the  integrity  of  the  research. 
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•  Submitted  local  status  and  technical  reports  and  recommendations  to  Wright 
Laboratory  regarding  accomplishments  and  future  direction. 


HI.  ENSURE  SITE  AND  EQUIPMENT  NEEDS  FOR  THE  RESEARCH  ARE 
MET 

•  Continued  to  provide  supervision  for  both  the  Trotwood-Madison  and  Dunbar 
F.A.S.T  Track  laboratories  to  effectively  meet  the  needs  of  the  ISIS  and 
Maestro  tutors. 


IV.  EFFECTIVELY  ADMINISTER  ALL  GRANT  FUNDS 

•  Flexibility  provided  by  contracting  for  technicians'  time  resulted  in  significant 
savings. 

•  Costs  to  prepare  school  labs  for  Year  Five  were  less  than  projected. 

•  During  this  extended  research  year  less  time  than  anticipated  was  required 
from  the  educational  consultants. 

•  Publication  and  reporting  costs  were  less  than  anticipated. 

•  No  major  maintenance  or  replacements  of  equipment  were  necessary  during 
Year  Five. 
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4.  Summaries  of  Years  One,  Two,  and  Three 

4. 1.  Summary  of  Year  One  (1992-1993) 


During  Year  One  of  the  project  computer  laboratories  were  constructed  at  Dunbar  and 
Trotwood-Madison  High  Schools.  Each  laboratory  contained  30  networked  NCR  486 
processor-based  computers.  Special  student  desks  were  designed  and  built  to  allow  for 
maximum  use  of  the  space  in  the  rooms  and  containment  of  wires  and  electrical  outlets. 
The  rooms  were  also  equipped  with  controllable  lighting  and  air  conditioning  to  allow 
for  maximum  comfort  and  continuity  between  the  test  sites. 

Armstrong  Laboratory  provided  the  labs  with  the  first  of  the  tutors  to  be  developed.  That 
tutor  was  called  the  WPS  (Word  Problem  Solving)  Tutor.  The  tutor  was  designed  to 
teach  ninth  grade  pre-algebra  students  word  problem  solving  skills  as  well  as  general 
problem  solving  strategies  with  broader  applications. 

Armstrong  Laboratory  conducted  research  to  compare  the  performance  of  students  who 
used  the  tutor  to  similar  students  who  did  not.  Armstrong  Laboratory  staff  has  published 
those  results. 

The  local  research  team  conducted  student  and  teacher  surveys  to  determine  the  general 
perception  of  the  tutor.  The  teachers  generally  felt  that  the  tutor  had  a  positive  impact  on 
both  their  students  and  their  own  teaching  abilities.  In  addition,  the  students  generally 
reported  a  positive  impression  of  the  tutor. 


4.2.  Summary  of  Year  Two  (1993-1994) 

During  Year  Two  of  the  project,  the  R-WISE  (Reading  and  Writing  in  a  Supportive 
Environment)  tutor  was  introduced  to  the  labs.  This  tutor  was  designed  to  improve  ninth 
grade  English  students’  reading  and  writing  skills  by  offering  computerized  tutoring 
specifically  adapted  to  each  student’s  performance  level  and  interests.  The  tutor  guided 
students  as  they  practiced  a  logical  sequence  of  steps  including  reading,  comprehension, 
pre-writing,  drafting,  revision,  and  editing. 

Armstrong  Laboratory  not  only  continued  to  conduct  research  on  the  WPS  tutor  but  also 
developed  a  series  of  research  questions  for  to  the  R-WISE  tutor.  For  the  R-WISE  tutor 
Armstrong  Laboratory  compared  the  performance  of  students  who  used  the  tutor  to 
students  who  used  a  simple  word  processor  for  an  equivalent  time  and  activity. 
Armstrong  Laboratory  staff  have  published  those  results. 
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The  local  research  team  conducted  a  series  of  focus  group  discussions  with  teachers  and 
students  involved  in  the  project.  Some  frustration  was  expressed  by  the  teachers  with 
regard  to  the  initial  difficulty  of  learning  the  tutor  and  trying  to  instruct  students  when 
the  teachers  were  not  totally  comfortable  with  the  system.  Both  students  and  teachers 
stated  that  the  tutors  had  a  positive  impact  on  the  learning  process. 


4.3.  Summary  of  Year  Three  (1994-1995) 

Originally,  Year  Three  was  scheduled  to  be  the  year  that  the  science-based  tutor  would 
be  implemented.  However,  delays  in  the  development  of  the  science-based  tutor  required 
that  implementation  of  that  tutor  be  delayed  by  a  year.  Therefore,  Year  Three  was 
termed  a  maintenance  year  for  the  two  previously  developed  tutors.  Minor  software 
upgrades  were  made  to  those  tutors,  but  no  new  activities  were  initiated. 

Armstrong  Laboratory  continued  to  gather  data  on  both  of  the  tutors.  Two  versions  of 
the  R-WISE  tutor  were  installed  and  compared  to  see  how  the  differences  impacted 
student  performance.  The  difference  in  the  tutors  consisted  primarily  of  the  ways  in 
which  the  students  were  given  help  during  a  session  on  the  computer.  One  version  of  the 
tutor  gave  the  students  very  directed  help  at  numerous  times  during  the  writing  process 
while  the  other  version  of  the  tutor  provided  help  only  when  it  was  solicited  by  the 
students.  Armstrong  Laboratory  will  publish  the  results  of  this  study. 

Locally,  the  research  activities  of  Year  Three  focused  on  the  effect  of  the  tutors  on 
student  ability  to  pass  the  Ohio  Proficiency  Test.  Passing  rates  and  scores  for  students 
who  did  and  did  not  use  the  tutors  were  compared.  No  significant  improvement  in 
student  pass  rate  of  the  math  section  of  the  Ohio  Proficiency  Test  was  seen  for  students 
who  used  the  WPS  tutor.  Students  who  used  the  R-WISE  tutor,  however,  showed  a 
significantly  higher  passing  rate  on  the  writing  section  of  the  Ohio  Proficiency  Test  than 
students  who  did  not  use  the  tutor. 
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5.  Year  Four  as  an  ISIS  Evaluation  Year 


5. 1.  Maintenance  Year  for  WPS  and  R-WISE  Tutors 

Year  Four  was  a  maintenance  year  for  both  the  WPS  and  the  R-WISE  tutors.  Armstrong 
Laboratory  continued  to  collect  data  from  the  English  and  mathematics  classes  using  the 
tutors  and  to  provide  minor  upgrades  to  the  software. 


5.2.  Year  Four  Research  Design  for  ISIS  tutor 

5.2.1.  Description  of  the  Tutor 

In  Year  Four  the  third  tutor,  ISIS  (Instmction  in  Science  Inquiry  Skills),  was 
implemented  at  Dunbar  and  Trotwood-Madison  High  Schools.  The  primary  goal  of  the 
ISIS  tutor  is  to  teach  students  the  cognitive  skills  underlying  the  principles  of  scientific 
inquiry.  Specifically,  students  will  understand  and  demonstrate  how  to  generate  a 
hypothesis,  design  an  experiment  to  test  that  hypothesis,  conduct  the  experiment  in  a 
simulated  ecosystem,  draw  conclusions  about  the  experiment,  and  accept  or  reject  the 
hypothesis  based  on  those  conclusions. 

The  second  goal  of  ISIS  is  to  address  introductory  high  school  biology  in  the  area  of 
ecology  concepts  and  relationships.  An  understanding  of  these  concepts  will  increase 
student  potential  for  becoming  scientifically  literate,  functional,  and  critical. 

There  are  approximately  45  major  domain  concepts  embedded  in  ISIS  including  biomes 
(e.g.,  grasslands,  deserts,  temperate  deciduous  forests),  biotic  factors  (e.g.,  carbon 
dioxide,  rainfall,  sunlight),  atmospheric  conditions  (e.g.,  greenhouse  effect,  pollutants), 
and  ecological  relationships  (e.g.,  succession,  symbiosis). 


5.2.2.  Research  Design 

There  were  no  “non-treatment”  group  classes  involved  in  the  computer  laboratories 
during  Year  Four.  Instead,  Armstrong  installed  two  versions  of  the  ISIS  tutor  and 
designed  a  series  of  research  questions  around  the  differences  in  these  two  versions. 
Essentially,  one  version  had  students  enter  into  an  idea  mapping  domain  upon  completion 
of  an  experiment.  In  that  domain,  the  students  were  asked  to  draw  cause  and  effect 
connections  between  ideas  related  to  their  problem.  The  other  version  of  the  tutor  simply 
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did  not  contain  this  domain.  Students  immediately  went  on  to  another  problem  when 
they  completed  an  assignement  associated  with  an  experiment. 


5.2.3.  Armstrong  Laboratory’s  Role  in  the  Evaluation 

The  project’s  primary  research  team,  Armstrong  Laboratory,  administered  a  series  of 
tests  including  attitudinal  surveys  and  subject  matter  tests  at  the  beginning  and  the  end  of 
the  school  year.  Armstrong  Laboratory  personnel  will  publish  their  results  when  they  are 
available. 


5.3.  Local  Quantitative  Research  Project 

In  cooperation  with  Armstrong  Laboratory  and  local  school  districts  the  local  research 
team  conducted  a  quantitative  research  project  during  Year  Four.  The  purpose  of  this 
research  activity  was  to  determine  the  effect,  if  any,  of  the  use  of  the  tutors  on  the  ability 
of  students  to  improve  their  performance  on  and/or  pass  related  sections  of  the  Ohio 
Proficiency  Test.  This  test  is  a  ninth-grade  level  test  administered  state-wide.  Passage  of 
the  test  is  required  to  receive  a  diploma  upon  graduation.  Data  relating  to  students  who 
used  either  the  WPS  or  the  R-WISE  tutors  during  the  1993-1994,  1994-1995,  or  1995- 
1996  school  years  were  included  in  the  study.  The  study  did  not  include  information 
about  students  who  used  the  ISIS  tutor  because  there  was  not  a  relevant  section  of  the 
Ohio  Proficiency  Test  to  use  as  an  evaluation  of  usage  of  that  tutor. 

The  local  research  team’s  report  is  attached  as  Appendix  A. 
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6.  Year  Five  as  a  Maestro  Evaluation  Year 

6. 1.  Maintenance  Year  for  WPS  and  ISIS  Tutors 

Year  Five  was  a  maintenance  year  for  both  the  WPS  and  the  ISIS  tutors.  Armstrong 
Laboratory  continued  to  collect  data  from  the  science  and  mathematics  classes  using  the 
tutors  and  to  provide  minor  upgrades  to  the  software. 


6.2.  Year  Five  Research  Design  for  the  Maestro  tutor 

6.2.1.  Description  of  the  Maestro  Tutor 

In  Year  Five  Armstrong  Laboratory  implemented  a  significantly  updated  version  of  the 
R-WISE  tutor  which  they  renamed  Maestro.  The  new  tutor  was  more  refined  and 
integrated  than  earlier  versions  of  R-WISE.  The  new  version  allowed  development  of 
ideas  and  writing  samples  to  be  expanded  within  the  tutor  and  used  in  subsequent 
sections  of  the  tutor  for  a  smoother  flow  of  ideas  during  the  writing  process. 


6.2.2.  ISIS  Research  Design 

There  were  no  “non-treatment”  group  classes  involved  in  the  computer  laboratories 
during  Year  Five. 


6.2.3.  Armstrong  Laboratory’s  Role  in  the  Evaluation 

The  project’s  primary  research  team,  Armstrong  Laboratory,  administered  a  series  of 
tests  including  attitudinal  surveys  and  subject  matter  tests  at  the  beginning  and  the  end  of 
the  school  year.  Armstrong  Laboratory  personnel  will  publish  their  results  when  they  are 
available. 


6.3.  Local  Quantitative  Research  Project 

No  local  research  activities  were  conducted  during  year  Five. 
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7.  Additional  Activities 


Examples  of  additional  programs  administered  by  the  Alliance  for  Education  and 
partially  funded  by  the  F.A.S.T.  Track  grant  are  Project  GEMMA,  Wright  Connection, 
and  f 


Project  GEMMA  ( Growth  in  Education  through  a  Mathematical/Scientific  Alliance ) 
was  designed  in  1990  to  provide  experiences  for  high  school  teachers  to  explore  science 
and  mathematics  in  the  world  beyond  the  classroom  walls.  Nearly  100  teachers  in 
Montgomery  and  Greene  counties  participated  in  summer  internships  at  local  business, 
industry  or  government  sites  where  they  actively  took  on  "real  world”  work  experiences 
designed  to  provide  meaningful  classroom  transfer  opportunities.  Supported  by  a 
partnership  of  representatives  from  Dayton-area  businesses,  Wright  Laboratory, 
schools/universities,  the  cornerstone  of  the  project  was  one-on-one  mentoring  at  local 
business  and  government  sites,  professional  development  seminars,  strategies  to  transfer 
lessons  learned  to  the  classroom  and  academic-year  dissemination  activities. 


Project  Wright  Connection  is  an  expansion  of  Project  GEMMA  which  began  in  1995 
when  the  National  Science  Foundation  awarded  over  $2  million  to  the  Alliance.  A 
partnership  among  the  Alliance,  Wright  Laboratory,  area  businesses,  the  Engineering  and 
Science  Foundation  and  Miami  Valley  schools,  Wright  Connection  now  provides 
opportunities  for  teachers  in  14  counties  to  participate.  The  project  continues  to  support 
meaningful  summer  internships  as  the  cornerstone  of  learning.  Many  of  the  placement 
sites  are  at  WPAFB. 

Grants  for  classroom  activities  provide  resources  for  educators  to  implement  new 
techniques  that  address  topics  such  as  critical  thinking,  technology,  scientific  process  and 
cooperative  learning.  Over  200  direct  participants  and  hundreds  of  other  school  team 
members  will  benefit  from  Wright  Connection  as  networks  are  established  and  teachers 
gain  the  skills  necessary  to  prepare  students  for  the  future  workplace. 
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I4  was  an  initiative  done  in  partnership  with  the  Disney  Celebration  Teaching  Academy 
to  research  Air  Force  technology  applications  in  K-12  schools.  Activities  involved 

•  Ascertaining  the  current  state-of-the-art  technology  used  in  education; 

•  Identifying  effective  technologies  currently  used  in  the  areas  of  mathematics, 
science  and  reading/writing/communications  education  that  can  be  advanced 
with  an  infusion  of  Air  Force  technology; 

•  Facilitating  the  initial  steps  of  the  infusion  process  by  communicating  the 
results  of  the  study  through  production  of  a  report  upon  the  completion  of 
research. 
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8.  APPENDIX  A  -  Ohio  Proficiency  Test  Report 


Fundamental  Skills  Tutor 
Correlation  with 
Ohio  Proficiency  Test  Data 
for  the  1993-1994,  1994-1995,  and  1995-1996 

School  Years 


Presented  to  the 
Alliance  for  Education 


January  10, 1997 


Prepared  by 
Katie  Thorp 

University  of  Dayton  Research  Institute 
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EXECUTIVE  SUMMARY 


Artificial  intelligence  based  software  programs  have  been  developed  by  the  US  Air 
Force's  Armstrong  Laboratory  to  aid  in  teaching  of  the  fundamental  skills  of  writing  and 
algebra  word  problem  solving.  These  tutors  have  been  field  tested  locally  at  Dayton 
Dunbar  High  School  and  Trotwood-Madison  High  School.  The  focus  of  this  research 
was  to  determine  the  effect  of  tutor  usage  on  student  performance  on  the  Ohio 
Proficiency  Test  (OPT). 

Each  of  the  tutors  was  addressed  separately;  and  in  each  case  the  study  population  was 
limited  to  students  who  failed  the  math  or  writing  portion  of  the  OPT  in  the  fall.  The 
comparison  of  subscores  between  groups  in  the  fall  was  then  used  to  validate  the 
statistical  equivalence  of  the  groups  at  the  beginning  of  the  school  year.  This  does  not, 
however,  mean  that  the  OPT  is  a  valid  test  to  measure  student  learning  with  the  tutors. 

There  was  a  lack  of  any  statistical  significance  in  the  passing  rate  on  the  OPT  of  the 
students  who  used  the  word  problem  solving  tutor  over  those  who  did  not,  indicating  that 
the  software  was  not  effective  at  helping  students  pass  this  important  exam.  There  was, 
however,  a  significant  improvement  in  passing  rate  for  students  who  did  not  use  the  tutor 
in  the  Workshop  classes  at  Trotwood-Madison  over  students  in  Workshop  classes  which 
did  use  the  tutor.  This  could  suggest  that  the  tutor  was  not  an  appropriate  teaching  aid 
for  this  group  of  students.  It  could,  however,  simply  be  a  reflection  of  the  inability  of 
OPT  data  to  reflect  learning  accomplished  through  the  use  of  the  tutor. 

Comparison  of  the  subscores  for  the  students  who  did  and  did  not  use  the  tutor  suggested 
that  the  only  area  which  showed  an  improvement  of  the  tutor  group  over  the  control 
population  was  in  the  one  area  not  believed  to  be  addressed  by  the  tutor  (Data  Analysis). 
These  results  further  substantiate  the  suggestion  that  the  word  problem  solving  tutor  is 
not  an  effective  teaching  aid  for  the  material  covered  in  the  Ohio  Proficiency  Test. 

Conversely,  the  results  for  the  writing  tutor  were  much  more  promising.  There  was  a 
significant  increase  in  the  passing  rate  for  the  students  who  used  the  writing  tutor  over 
those  who  did  not.  This  strongly  indicates  that  the  writing  tutor  software  was  highly 
effective  at  helping  students  pass  this  important  exam. 
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Comparison  of  the  subscores  for  the  groups  shows  that  the  tutor  group  was  able  to 
improve  their  relative  score  from  the  lowest  to  the  highest  in  the  area  of  Content 
Organization.  Likewise,  the  tutor  group  was  able  to  at  least  remain  equivalent  to  the 
other  groups  in  the  other  subscore  areas.  The  greatest  help  appears  to  be  in  the  area  of 
Content  Organization  which  is  the  area  most  strongly  addressed  by  the  tutor. 

Of  course,  other  factors  not  controlled  by  the  researchers  can  always  affect  the  observed 
results.  The  fact  that  a  large  percentage  of  the  control  population  came  from  another 
school  may  be  a  cause  for  concern.  Likewise,  differences  in  school  curriculums,  student 
populations,  teacher  experience  and  style,  and  mid-year  updates  of  the  tutors  themselves 
could  all  play  a  part  in  affecting  the  results. 

All  of  this  analysis  must  be  considered  with  the  understanding  that  the  Ohio  Proficiency 
Test  was  not  designed  to  be  used  as  a  measure  of  instructional  techniques.  In  addition,  a 
number  of  variables  were  not  controlled  in  this  comparison  and  could  have  had  a 
significant  impact  on  the  results  and  conclusions.  Furthermore,  both  tutors  address  a 
variety  of  material  not  evaluated  by  the  OPT  which  could  be  of  value  to  the  students. 
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1.  Background 


Several  artificial  intelligence-based  software  tutoring  programs  have  been  developed  by 
the  US  Air  Force's  Armstrong  Laboratory  to  aid  in  teaching  the  fundamental  skills  of 
writing  (R-WISE),  science  (ISIS),  and  algebra  word  problem  solving  (WPS).  Each  of 
these  tutors  has  been  field  tested  at  Dayton  Dunbar  High  School  and  Trotwood-Madison 
High  School  as  well  as  several  other  sites  across  the  nation.  The  design  of  the  R-WISE 
and  WPS  tutors  suggests  they  may  help  students  increase  scores  and  pass  the  Ohio 
Proficiency  Test  (OPT).  Through  correlation  of  student  usage  of  the  tutor  with  OPT 
results,  local  researchers  hoped  to  determine  if  the  tutors  were  indeed  an  advantageous 
tool  for  preparation  for  this  important  exam.  However,  it  should  be  noted  that  the  tutors 
and  Ohio  Proficiency  Test  were  not  designed  for  use  together.  A  lack  of  an  improvement 
in  OPT  scores  does  not  necessarily  indicate  that  the  students  did  not  learn  as  a  result  of 
using  the  tutor.  It  could  simply  mean  that  the  OPT  is  not  the  appropriate  method  of 
measurement  for  the  effects  of  the  tutors. 

Ohio  Proficiency  testing  in  the  field  of  science  was  not  initiated  until  the  spring  of  1996 
and  as  result,  comparison  of  tutor  usage  and  student  performance  was  not  posibile  for 
that  tutor. 


2.  The  Word  Problem  Solving  Tutor/OPT  Analysis 

This  analysis  includes  data  collected  from  Dayton  Dunbar  High  School  and  Trotwood- 
Madison  High  School  during  the  1993-1994,  1994-1995,  and  1995-1996  school  years. 


2. 1.  The  Word  Problem  Solving  Tutor 

The  WPS  tutor  is  divided  into  a  series  of  modules  which  address  various  topics.  These 
modules  were  designed  to  be  appropriate  for  the  class  materials  covered  in  a  general  9th 
grade  Algebra  course. 
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2.2.  The  Math  Section  of  the  OPT 


The  Ohio  Department  of  Education  has  established  a  series  of  strands  and  learning 
outcomes  which  the  math  section  of  the  Ohio  Proficiency  Test  has  been  designed  to 
measure.  These  strands  and  learning  outcomes  are  listed  in  Table  1  and  identified  as  to 
whether  there  are  WPS  tutor  modules  which  address  similar  topics.  Additional 
information  about  the  mathematics  learning  outcomes  is  available  in  information 
published  by  the  Ohio  Department  of  Education. 

Examples  of  test  results  from  the  math  section  of  the  Ohio  Proficiency  Test  are  given  in 
Table  2.  Each  section  of  the  Ohio  Proficiency  Test  is  scored  on  three  levels.  The  first 
level  is  a  pass/fail  marking.  A  student  who  passes  the  test  is  not  given  any  further 
scoring  (i.e.  score  or  subscores  are  omitted),  and  only  a  passing  mark  is  returned  to  the 
school.  A  student  who  fails  the  math  section  of  the  test  is  then  given  a  general  score 
which  can  range  from  0  to  199  given  to  three  significant  figures  (i.e.  176,  180,...).  The 
student  is  then  given  a  subscore  ranking  in  each  of  the  five  following  strand  areas: 
Arithmetic,  Measurement,  Geometry,  Data  Analysis,  and  Algebra.  The  possible  subscore 
rankings  are  +,  *,  or  -  as  defined  in  Table  2. 

Through  analysis  of  the  goals  and  directives  of  the  WPS  tutor  and  the  Ohio  Proficiency 
Test,  it  was  determined  by  the  researchers  that  the  WPS  tutor  did  not  directly  address  the 
issues  evaluated  in  the  Data  Analysis  subscore  area.  Therefore,  the  tutor  would  not  be 
expected  to  increase  student  performance  in  this  area.  The  other  areas,  however,  were 
believed  to  be  addressed  by  the  tutor  and  should  be  affected  by  tutor  usage. 


2.3.  Study  Populations 

The  study  population  included  only  those  students  who  failed  the  math  section  of  the 
Ohio  Proficiency  Test  in  the  fall  of  the  year  of  study  and  for  whom  data  was  available  for 
the  spring  test  of  that  same  year.  Data  were  collected  for  the  1993-1994, 1994-1995,  and 
1995-1996  school  years.  The  sample  size  for  each  population  group  and  year  are  given 
in  Table  3. 
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Table  1:  WPS  tutor  modules  and  OPT  mathematics  strands. 


OPT  MATHEMATICS  STRANDS 


Arithmetic 

1.  Compute  with  whole  numbers,  fractions,  and  decimals 

2.  Compare,  order,  and  determine  equivalence  of  fractions, 
decimals,  percents,  whole  numbers,  and  integers 

3.  Solve  and  use  proportions 

4.  Round  numbers  to  the  nearest  thousand,  hundred,  ten,  one, 
tenth,  and  hundredth 

5.  Solve  problems  and  make  applications  involving  percentages 


Measurement 

6.  Select  and  compute  with  appropriate  standard  or  metric  units 
to  measure  length,  area,  volume,  angles,  weight,  capacity, 
time,  temperature,  and  money 

7.  Convert,  compare,  and  compute  with  common  units  of 
measure  within  the  same  measurement  system 

8.  Read  the  scale  on  a  measurement  device  to  the  nearest  mark 
and  make  interpolations  where  appropriate 


Geometry 

9.  Recognize,  classify,  and  use  characteristics  of  lines  and  simple 
two-dimensional  figures 

10.  Find  the  perimeters  (circumference)  and  areas  of  polygons 
(circles) 

1 1 .  Find  surface  areas  and  volumes  of  rectangular  solids 


Data  Analysis 

12.  Read,  interpret,  and  use  tables,  charts,  maps,  and  graphics  to 
identify  patterns,  note  trends,  and  draw  conclusions 

13.  Use  elementary  notions  of  probability 

14.  Compute  averages 


Algebra 

15.  Solve  simple  number  sentences  and  use  formulas 

16.  Evaluate  algebraic  expressions  (simple  substitutions) 


FOCUS  OF 


WPS  TUTOR 


1. 

Yes 

2. 

No 

3. 

Yes 

4. 

Yes 

5. 

Yes 

6. 

No 

7. 

Yes 

8. 

No 

9. 

No 

10. 

Yes 

11. 

Yes 

12. 

No 

13. 

No 

14. 

No 

15. 

Yes 

16. 

Yes 
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Table  2:  Sample  results  for  the  math  section  of  the  Ohio  Proficiency  Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

MEASURE¬ 

MENT 

ARITH¬ 

METIC 

GEOM¬ 

ETRY 

DATA 

ANALYSIS 

ALGE¬ 

BRA 

Larry 

Student 

Pass 

Mary 

Student 

Fail 

182 

+ 

— 

* 

— 

Carry 

Student 

Fail 

176 

— 

— 

— 

+ 

* 

Berry 

Student 

Fail 

196 

+ 

+ 

* 

* 

-  =  Performance  lower  than  expected  of  students  at  the  standard 
*  =  Performance  approximately  the  same  as  expected  of  students  at  the  standard 
+  =  Performance  higher  than  expected  of  students  at  the  standard 


Table  3:  Number  of  students  included  in  the  WPS  data  analysis. 


WPS 

Control 

Total 

Dunbar 

Algebra 

T-M 

Algebra 

T-M 

Workshop 

Dunbar 

Algebra 

T-M 

Algebra 

T-M 

Workshop 

93-94 

School 

Year 

82 

55 

20 

33 

0 

0 

190 

94-95 

School 

Year 

38 

37 

72 

103 

53 

52 

355 

95-96 

School 

Year 

71 

0 

37 

103 

0 

55 

266 

mWhiB 

191 

92 

129 

239 

53 

107 

w!Wl 

283 

129 

292 

107 

811 
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2.3.1.  Control 


The  control  population  included  students  who  were  enrolled  in  an  algebra  class  which  did 
not  use  the  Air  Force-developed  software  designed  to  enhance  word  problem  solving 
proficiency.  The  control  population  would  include  students  who  were  operating  at  class 
level,  above  class  level,  and  below  class  level.  For  the  purposes  of  this  study,  the  control 
population  was  assumed  to  be  equivalent  to  the  treatment  population.  This  assumption 
was  enhanced  by  the  fact  that  the  study  population  was  limited  to  students  who  failed  the 
math  portion  of  the  Ohio  Proficiency  Test  during  the  fall  of  the  year  in  question. 


2.3.2.  WPS 

The  WPS  population  included  students  from  Dunbar  and  Trotwood-Madison  High 
Schools  who  were  enrolled  in  an  algebra  class  which  used  the  WPS  tutor.  This  group 
represents  the  experimental  population  in  this  study. 


2.4.  Data  Analysis 

In  all  cases,  a  Chi-Square  analysis  with  a  confidence  interval  of  5%  was  used  to 
determine  the  statistical  significance  of  the  data.  This  indicates  that  if  the  numbers  are 
statistically  significant,  then  there  is  at  least  a  95%  certainty  that  the  same  statistics 
would  be  observed  in  other,  equivalent  populations.  Tabular  representations  of  all  of  the 
data  presented  are  included  in  Appendix  Al. 

For  this  analysis  all  of  the  treatment  groups  (Dunbar  Algebra  I  Part  I,  Trotwood  Practical 
Algebra,  and  Trotwood  Workshop)  were  combined  into  one  large  treatment  group,  and 
likewise  their  representative  control  groups  were  combined  into  one  large  control  group. 
This  gave  the  largest  sample  size  possible  and  should  provide  the  most  accurate  statistics. 
Combining  the  groups  should  also  show  the  effectiveness  of  using  the  tutor  in  general, 
with  less  affect  arising  from  school  curriculum  differences,  teacher  styles,  and  class 
structure  differences. 
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2.4.1.  Comparison  of  Data  Between  Groups 


By  combining  the  data  according  to  group,  regardless  of  the  year  of  study,  a  comparison 
can  be  made  between  the  groups.  This,  then,  can  be  used  to  determine  the  effectiveness 
of  the  tutor. 


2.4.1.L  Fall 

2.4.1 .1 .1 .  Pass/Fail  Analysis 

Due  to  the  fact  that  the  study  population  was  limited  to  students  who  failed  the  OPT  in 
the  fall,  no  comparison  can  be  made  in  this  area. 

2.4.1 .1 .2.  Score  Analysis 

Due  to  the  nearly  continuous  spread  in  the  scores  for  the  math  section  of  the  OPT,  this 
analysis  is  complex  and  is  not  available  at  this  time. 

2.4.1 .1 .3.  Subscore  Analysis 

Chi-Squared  analysis  of  fall  subscores  showed  no  statistically  significant  variance  in  the 
groups.  This  indicates  that  the  treatment  and  control  groups  were  statistically  equivalent 
at  the  beginning  of  the  year.  This  enhances  the  validity  of  our  comparison  between  these 
two  groups. 


2.4.I.2.  Spring 

2.4.1 .2.1 .  Pass/Fail  Analysis 

This  measure  of  the  data  gives  the  clearest  representation  of  the  effectiveness  of  the  tutor 
at  helping  students  pass  the  OPT.  The  goal  of  each  student  is  to  pass  the  test.  An 
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increase  in  score  may  suggest  an  increase  in  ability,  but  without  passing  the  test  the 
students  and  schools  do  not  measure  any  great  improvement.  When  all  three  school 
years  were  combined,  no  statistically  significant  difference  was  measured  between  the 
passing  rate  of  students  using  the  tutor  and  those  not  using  the  tutor. 

This  suggests  that  use  of  the  WPS  tutor  had  no  affect  on  student  passing  rates  for  the 
math  section  of  the  OPT.  The  lack  of  a  difference  in  the  passing  rates  could  easily  be 
attributed  to  other  variables  such  as  teacher  differences,  school  curriculum  variations, 
student  population  differences,  and  other  non-controlled  variables. 

Similar  comparisons  were  also  made  for  each  of  the  schools  and  courses  independently. 
These  comparisons  again  showed  a  lack  of  significance. 

2.4.1. 2.2.  Score  Analysis 

Due  to  the  nearly  continuous  spread  in  the  scores  for  the  math  section  of  the  OPT,  this 
analysis  is  complex  and  is  not  available  at  this  time. 

2.4.1 .2.3.  Subscore  Analysis 

The  Chi-Squared  analysis  of  spring  subscores  showed  a  statistically  significant  variance 
only  in  the  subscore  category  of  Algebra.  The  other  subscore  categories  did  not  show  any 
significant  difference  between  the  treatment  and  the  control  populations.  The  area  of 
algebra  was  believed  to  be  addressed  by  the  tutor,  however,  students  who  did  not  use  the 
tutor  appeared  to  do  better  in  this  subscore  area.  This  could  indicate  that  either  the  initial 
tutor  goals  were  not  properly  identified,  the  OPT  is  measuring  another  difference 
between  the  study  populations,  or  the  tutor  is  not  helping  students  in  this  area  and  the  lost 
class-time  is  actually  detrimental. 


2.5.  Results  of  WPS  Tutor  Analysis 

Although  a  lot  of  data  were  collected  and  analyzed,  the  most  relevant  question  to  be 
answered  relates  to  the  effectiveness  of  use  of  the  tutor  to  aid  students  with  passing  the 
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OPT.  The  fact  that  the  study  population  was  limited  to  students  who  failed  the  math 
portion  of  the  OPT  in  the  fall  and  that  the  comparison  of  subscores  between  groups  in  the 
fall  suggest  that  the  groups  were  statistically  equivalent  at  the  beginning  of  the  school 
year  suggest  that  this  statistical  comparison  between  students  who  did  and  did  not  use  the 
tutor  should  be  valid.  It  does  not,  however,  mean  that  the  OPT  is  a  valid  test  to  measure 
student  learning  with  WPS.  It  will  simply  be  an  indication  of  whether  students  who  used 
the  tutor  performed  better  on  the  OPT. 

The  lack  of  any  statistical  significance  in  the  passing  rate  of  the  students  who  used  the 
WPS  tutor  over  those  who  did  not  indicates  that  the  WPS  software  may  not  be  effective 
at  helping  students  pass  this  important  exam.  It  could  also,  however,  simply  be  a 
reflection  of  the  inability  of  OPT  data  to  reflect  learning  accomplished  through  the  use  of 
the  tutor. 

Of  course,  other  factors  not  controlled  by  the  researchers  can  always  affect  the  observed 
results.  Differences  in  school  curriculums,  student  populations,  teacher  experience  and 
style,  and  mid-year  updates  of  the  tutors  themselves  could  all  play  a  part  by  affecting  the 
data  analysis. 


3.  The  R-WISE  Tutor/OPT  Analysis 

This  analysis  includes  data  collected  from  Dayton  Dunbar  High  School,  Dayton  Belmont 
High  School,  and  Trotwood-Madison  High  School  during  the  1993-1994,  1994-1995, 
and  1995-1996  school  years. 

3.1.  The  R-WISE  Tutor 

The  R-WISE  tutor  (Reading  and  Writing  in  a  Supportive  Environment)  is  divided  into  a 
series  of  tools  which  address  various  topics.  Through  observation  and  use  of  the  tutor 
and  analysis  of  some  of  the  literature  provided  by  Armstrong  Laboratory  on  the 
objectives  of  the  tutor,  correlations  were  made  between  the  writing  characteristics  and 
learning  outcomes  defined  by  the  Ohio  Department  of  Education  for  the  OPT  and  the 
tutor  itself.  This  comparison  is  shown  in  Table  4. 
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3.2.  The  Writing  Section  of  the  OPT 


The  Ohio  Department  of  Education  has  established  a  series  of  characteristics  and  learning 
outcomes  which  the  writing  section  of  the  Ohio  Proficiency  Test  has  been  designed  to 
measure.  Table  4  lists  the  writing  characteristics  and  related  learning  outcomes  as 
defined  by  the  Ohio  Department  of  Education.  During  the  1995-1996  school  year  the 
area  of  Content/Organization  was  divided  into  two  sections,  making  comparison  between 
the  three  years  inconsistent. 


Table  4:  OPT  writing  characteristics  and  R-WISE  tutor  objectives. 


OPT  WRITING  CHARACTERISTICS 

FOCUS  OF 
R-WISE  TUTOR 

Content/Oreanization 

1.  Conveys  a  message  related  to  the  prompt 

1.  Yes 

2.  Includes  supporting  ideas  or  examples 

2.  Yes 

3.  Follows  a  logical  order 

3.  Yes 

4.  Conveys  a  sense  of  completeness 

4.  Yes 

Language 

5.  Exhibits  word  choice  appropriate  to  the  audience, 

5.  Yes 

purpose,  and  subject 

6.  No 

6.  Includes  clear  language 

Writing  Conventions 

7.  Contains  complete  sentences  and  may  contain 

7.  No 

purposeful  fragments 

8.  No 

8.  Exhibits  subject-verb  agreement 

9.  Contains  standard  forms  of  verbs  and  nouns 

9.  No 

10.  Exhibits  appropriate  punctuation 

10.  No 

1 1 .  Exhibits  appropriate  capitalization 

11.  No 

12.  Contains  correct  spelling 

12.  No 

13.  Is  legible 

13.  No 
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Examples  of  possible  test  results  from  the  writing  section  of  the  Ohio  Proficiency  Test 
are  given  in  Table  5.  The  writing  section  of  the  OPT  is  scored  on  three  levels.  The  first 
level  is  a  pass/fail  marking.  A  student  who  passes  the  test  is  not  given  any  further 
scoring  (i.e.  score  or  subscores  are  omitted),  and  only  a  passing  mark  is  returned  to  the 
school.  A  student  who  fails  the  writing  section  of  the  test  is  then  given  a  general  score 
which  can  range  from  0  to  4.5  given  in  two  significant  figures  (i.e.  3.5, 4.0,...).  The 
student  is  then  given  a  subscore  ranking  in  each  of  the  three  following  characteristic 
areas:  Content/Organization,  Language,  and  Writing  Conventions.  The  possible 
subscore  rankings  are:  satisfactory,  needs  some  help,  or  needs  help. 


Table  5:  Sample  results  for  the  writing  section  of  the  Ohio  Proficiency  Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

ORGANIZATION 

LANGUAGE 

WRITING 

CONVENTIONS 

Larry 

Student 

Pass 

Mary 

Student 

Fail 

4.0 

NH 

NSH 

S 

Carry 

Student 

Fail 

3.5 

NSH 

NH 

NH 

Berry 

Student 

Fail 

NH 

NH 

S 

S  =  Satisfactory 

NSH  =  Needs  Some  Help 

NH  =  Needs  Help 


Through  analysis  of  the  goals  and  directives  of  the  R-WISE  tutor  and  the  Ohio 
Proficiency  Test,  it  was  determined  by  the  researchers  that  the  R-WISE  tutor  did  not 
directly  address  the  issues  evaluated  in  the  Writing  Conventions  subscore  area. 
Therefore,  the  tutor  would  not  be  expected  to  increase  student  performance  in  this  area. 
The  other  areas,  however,  were  believed  to  be  addressed  by  the  tutor  and  should  be 
affected  by  tutor  usage. 
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3.3.  Study  Populations 


The  study  population  included  only  those  students  who  failed  the  writing  section  of  the 
Ohio  Proficiency  Test  in  the  fall  of  the  year  of  study  and  for  whom  data  were  available 
for  the  spring  test  of  that  same  year.  Data  were  collected  for  the  1993-1994,  1994-1995, 
and  1995-1996  school  years.  The  sample  size  for  each  population  group  and  year  are 
given  in  Table  6. 


Table  6:  Number  of  students  included  in  data  analysis  for  each  group  and  year  for 
R-WISE  analysis. 


R-WISE 

Dunbar 

R-WISE 

T-M 

WRITE 

Dunbar 

WRITE 

T-M 

Control 

Belmont 

Control 

T-M 

Total 

93-94 

School 

Year 

25 

44 

19 

12 

53 

42 

195 

94-95 

School 

Year 

36 

111 

0 

0 

77 

0 

224 

95-96 

School 

Year 

93 

96 

0 

0 

0 

0 

189 

School 

Total 

154 

251 

19 

12 

130 

42 

Group 
Total  ;.f 

405 

31 

172 

608 

3.3.1.  Control 

The  control  population  included  students  who  were  not  exposed  to  any  Air  Force- 
developed  software  designed  to  enhance  writing  proficiency.  The  majority  of  this 
population  were  students  in  the  ninth  grade  at  Belmont  High  School.  However,  some 
students  in  the  control  population  were  ninth  grade  students  at  Trotwood-Madison  High 
School  during  the  93-94  school  year  who  were  not  enrolled  in  a  ninth  grade  English  class 
which  used  the  tutor.  Because  class  rosters  were  not  always  available  for  classes  which 
did  not  use  the  tutor,  the  remaining  9th  grade  population  was  used.  As  a  result,  students 
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who  were  enrolled  in  a  class  other  than  a  standard  9th  grade  English  class  would  be 
included  in  this  population  group.  This  could  include  students  who  were  in  honors  level 
or  remedial  level  courses.  In  either  case,  the  control  population  would  include  students 
who  were  operating  at  class  level,  above  class  level,  and  below  class  level.  For  the 
purposes  of  this  study,  the  control  population  was  assumed  to  be  equivalent  to  the 
treatment  population.  The  accuracy  of  this  assumption  can  be  validated  by  comparison 
of  fall  scores  and  subscores  on  the  Ohio  Proficiency  Test.  This  assumption  is  further 
validated  by  the  fact  that  the  study  population  was  limited  to  students  who  failed  the 
writing  portion  of  the  Ohio  Proficiency  Test  during  the  fall  of  the  year  in  question. 


3.3.2.  WRITE 

The  WRITE  population  included  students  from  Dunbar  and  Trotwood-Madison  High 
Schools  who  were  enrolled  in  a  standard  9th  grade  English  class  which  used  a  simple 
word  processor  to  assist  in  writing.  The  software  did  not  contain  any  of  the  tutor 
assistance  available  in  the  more  advanced  version  of  the  tutor.  This  group  was  used  to 
study  the  effect  of  technology  usage  alone  compared  to  the  use  of  the  more  advanced 
tutor. 


3.3.3.  R-WISE 

The  R-WISE  population  included  students  from  Dunbar  and  Trotwood-Madison  High 
Schools  who  were  enrolled  in  a  standard  9th  grade  English  class  which  used  the 
advanced  R-WISE  tutor.  This  group  represents  the  true  experimental  population  in  this 
study. 


3.4.  Data  Analysis 

In  all  cases,  a  Chi-Square  analysis  with  a  confidence  interval  of  5%  was  used  to 
determine  the  statistical  significance  of  the  data.  This  indicates  that  if  the  numbers  are 
statistically  significant,  then  there  is  at  least  a  95%  certainty  that  the  same  statistics 
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would  be  observed  in  other,  equivalent,  populations.  Tabular  representations  of  all  of  the 
data  presented  are  included  in  Appendix  Al. 


3.4.1.  Comparison  of  Data  Between  Groups 

By  combining  the  data  according  to  group,  regardless  of  the  year  of  study,  a  comparison 
can  be  made  between  the  groups.  This,  then,  can  be  used  to  determine  the  effectiveness 
of  the  tutor. 


3.4.L1.  Fall 

3.4.1 .1 .1 .  Pass/Fail  Analysis 

Due  to  the  fact  that  the  study  population  was  limited  to  students  who  failed  the  OPT  in 
the  fall,  no  comparison  can  be  made  in  this  area. 

3.4.1 .1 .2.  Score  Analysis 

The  Chi-Squared  analysis  of  fall  scores  showed  no  statistically  significant  variance  in  the 
data. 


3.4. 1 . 1 .3.  Subscore  Analysis 

Comparison  of  the  fall  subscores  between  groups  could  not  be  made  in  the  area  of 
Content  Organization  because  of  a  change  in  the  way  the  test  was  scored  in  the  1995- 
1996  school  year.  The  other  subscore  areas  showed  that  the  groups  were  not  statistically 
equivalent  in  the  area  of  Writing  Conventions. 

In  the  area  of  Writing  Conventions  the  R-WISE  group  appeared  to  perform  the  best  prior 
to  exposure  to  the  tutor. 
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3. 4.1. 2.  Spring 

3.4.1 .2.1 .  Pass/Fail  Analysis 

This  measure  of  the  data  gives  the  clearest  representation  of  the  effectiveness  of  the  tutor 
at  helping  students  pass  the  OPT.  The  goal  of  each  student  is  to  pass  the  test.  An 
increase  in  score  may  suggest  an  increase  in  ability,  but  without  passing  the  test  the 
students  and  schools  do  not  measure  any  great  improvement.  No  significant  difference 
was  measured  in  the  passing  rate  for  students  who  either  did  or  did  not  use  the  tutor. 

3.4.1. 2.2.  Score  Analysis 

The  Chi-Squared  analysis  of  spring  scores  showed  no  statistically  significant  variance  in 
the  data. 


3.4.1. 2.3.  Subscore  Analysis 

Comparison  of  spring  subscores  by  group  showed  statistical  significance  only  in  the  area 
of  Writing  Conventions.  In  the  fall  subscore  comparison  of  these  two  groups,  they  were 
not  statistically  equivalent  with  the  R-WISE  group  performing  slightly  better.  After 
exposure  to  computer-based  training,  this  trend  was  still  apparent.  This  area  is  not 
believed  to  be  addressed  by  the  tutor.  Therefore,  if  the  tutor  is  affecting  scores,  we 
would  not  expect  it  to  affect  them  in  this  area 


3.5.  Results  of  R-WISE  Data  Analysis 

Although  a  lot  of  data  were  collected  and  analyzed,  the  most  relevant  question  to  be 
answered  relates  to  the  effectiveness  of  use  of  the  tutor  to  aid  students  with  passing  the 
OPT.  The  fact  that  the  study  population  was  limited  to  students  who  failed  the  writing 
portion  of  the  OPT  in  the  fall  and  that  the  comparison  of  subscores  between  groups  in  the 
fall  showed  no  significant  difference  between  groups,  suggesting  that  the  groups  were 
statistically  equivalent  at  the  beginning  of  the  school  year.  The  lack  of  a  significant 
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change  in  the  passing  rate  of  the  students  who  used  the  R-WISE  tutor  over  those  who  did 
not  suggests  that  the  tutor  is  not  enhancing  the  ability  of  students  to  pass  the  OPT. 

Of  course,  other  factors  not  controlled  by  the  researchers  can  always  affect  the  observed 
results.  The  fact  that  a  large  percentage  of  the  control  population  came  from  another 
school  may  be  a  cause  for  concern.  Likewise,  differences  in  school  curriculums,  student 
populations,  teacher  experience  and  style,  and  mid-year  updates  of  the  tutors  themselves 
could  all  play  a  part  in  affecting  the  data  analysis. 


4.  Conclusions 


It  appears  that  neither  of  the  tutors  is  increasing  student  performance  on  the  Ohio 
Proficiency  Test.  This  analysis  must  be  considered  with  the  understanding  that  the  Ohio 
Proficiency  Test  was  not  designed  to  be  used  as  a  measure  of  FST  instructional 
techniques.  In  addition,  a  number  of  variables  were  not  controlled  in  this  comparison 
and  could  have  had  a  significant  impact  on  the  results  and  conclusions.  Furthermore, 
both  of  the  tutors  address  material  which  is  not  evaluated  by  the  OPT  and  which  may  of 
value  to  the  students. 
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5. 


Appendix  A1  -  Tabulated  Data 


WPS  TUTOR 


Comparison  of  Data  by  Year 
Fall  Subscore  Data 


Measurement 


1993-1994 

1994-1995 

Total 

— 

154 

285 

439 

* 

34 

65 

99 

+ 

2 

5 

7 

Total 

190 

355 

545 

Arithmetic 


1993-1994 

1994-1995 

Total 

— 

155 

285 

440 

* 

32 

59 

91 

+ 

3 

11 

14 

Total 

190 

355 

545 

Geometry 


1993-1994 

1994-1995 

Total 

— 

53 

180 

233 

* 

118 

148 

266 

+ 

19 

27 

46 

Total 

190 

355 

545 
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Data  Analysis 


1993-1994 

1994-1995 

Total 

— 

113 

210 

323 

* 

69 

125 

194 

+ 

8 

20 

28 

Total 

190 

355 

545 

Algebra 


1993-1994 

1994-1995 

Total 

— 

110 

228 

338 

* 

70 

119 

189 

+ 

10 

8 

18 

Total 

190 

355 

545 

Spring  Pass/Fail  Data 


1993-1994 

1994-1995 

Total 

Pass 

22 

60 

82 

Fail 

163 

295 

458 

Total 

185 

355 

540 

Spring  Subscore  Data 


Measurement 


1993-1994 

1994-1995 

Total 

_ 

113 

195 

308 

* 

47 

91 

138 

+ 

3 

9 

12 

Total 

163 

295 

458 
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Arithmetic 


Geometry 


1993-1994 

1994-1995 

Total 

- 

57 

132 

189 

* 

86 

135 

221 

+ 

19 

28 

47 

Total 

162 

295 

457 

Data  Analysis 


1993-1994 

1994-1995 

Total 

- 

93 

208 

301 

* 

62 

85 

147 

+ 

7 

2 

9 

Total 

162 

295 

457 

Algebra 


1993-1994 

1994-1995 

Total 

- 

105 

118 

223 

* 

57 

159 

216 

+ 

0 

18 

18 

Total 

162 

296 

457 

37 


38 


Algebra 


WPS 

Control 

Total 

- 

183 

155 

338 

* 

110 

79 

189 

+ 

11 

7 

18 

Total 

304 

241 

545 

Spring  Pass/Fail  Data 


WPS 

Control 

Total 

Pass 

39 

43 

82 

Fail 

260 

198 

458 

Total 

299 

241 

540 

Spring  Subscore  Data 


Measurement 


WPS 

Control 

Total 

- 

177 

131 

308 

* 

79 

59 

U> 

00 

+ 

4 

8 

12 

Total 

260 

198 

458 

Arithmetic 


WPS 

Control 

Total 

- 

213 

156 

369 

* 

40 

33 

73 

+ 

6 

9 

15 

Total 

259 

198 

457 

39 


Geometry 


WPS 

Control 

Total 

_ 

106 

83 

189 

* 

122 

99  1 

221 

+ 

31 

16 

47 

Total 

249 

198 

457 

Data  Analysis 


WPS 

Control 

Total 

_ 

160 

141 

301 

* 

90 

57 

147 

+ 

9 

0 

9 

Total 

259 

198 

457 

Algebra 


WPS 

Control 

Total 

_ 

137 

86 

223 

* 

110 

106 

216 

+ 

12 

6 

18 

Total 

259 

198 

457 

R-W1SE  TUTOR 

Comparison  of  Data  by  Year 
Fall  Score  Data 


1993-1994 

1994-1995 

Total 

2 

6 

7 

13 

3 

26 

54 

80 

4 

163 

163 

326 

Total 

195 

224 

419 

40 


Fall  Subscore  Data 


Content  Organization 


1993-1994 

1994-1995 

Total 

NH 

161 

188 

349 

NSH 

34 

36 

70 

1  s 

0 

0 

0 

Total 

195 

224 

419 

Language 


1993-1994 

1994-1995 

Total 

NH 

136 

141 

277 

NSH 

44 

63 

107 

S 

15 

20 

35 

Total 

195 

224 

419 

Writing  Conventions 


1993-1994 

1994-1995 

Total 

NH 

120 

128 

248 

NSH 

42 

53 

95 

S 

33 

43 

76 

Total 

195 

224 

419 

Spring  Pass/Fail  Data 


1993-1994 

1994-1995 

Total 

Pass 

58 

78 

136 

Fail 

137 

146 

283 

Total 

195 

224 

419 

41 


Spring  Score  Data 


1993-1994 

1994-1995 

Total 

2 

7 

8 

15 

3 

35 

48 

83 

4 

95 

90 

185 

Total 

137 

146 

283 

Spring  Subscore  Data 


Content  Organization 


1993-1994 

1994-1995 

Total 

NH 

125 

106 

231 

NSH 

12 

38 

50 

S 

0 

0 

0 

Total 

137 

144 

281 

Language 


1993-1994 

1994-1995 

Total 

NH 

97 

71 

168 

NSH 

26 

48 

74 

S 

14 

25 

39 

Total 

137 

144 

281 

Writing  Conventions 


1993-1994 

1994-1995 

Total 

NH 

76 

74 

150 

NSH 

29 

31 

60 

S 

32 

39 

71 

Total 

137 

144 

281 

42 


Comparison  of  Data  by  Group 


Fall  Score  Data 


R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

2 

1 

10 

13 

3 

43 

1 

36 

80 

4 

171 

29 

126 

326 

Total 

216 

31 

172 

419 

Fall  Subscore  Data 


Content  Organization 


R-WISE 

WRITE 

CONTROL 

TOTAL 

!  NH 

190 

22 

137 

349 

NSH 

26 

9 

35 

70 

S 

0 

0 

0 

0 

Total 

216 

31 

172 

419 

Language 


R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

137 

20 

120 

277 

NSH 

58 

11 

38 

107 

S 

21 

0 

14 

35 

Total 

216 

31 

172 

419 

Writing  Conventions 


R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

114 

24 

110 

248 

NSH 

53 

4 

38 

95 

S 

49" 

3 

24 

76 

Total 

216 

31 

172 

419 

43 


Spring  Pass/Fail  Data 


R-WISE 

WRITE 

CONTROL 

TOTAL 

Pass 

84 

7 

45 

136 

Fail 

132 

24 

127 

283 

Total 

216 

31 

172 

419 

Spring  Score  Data 


R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

3 

0 

12 

15 

3 

36 

7 

40 

83 

4 

93 

17 

75 

185 

Total 

132 

24 

127 

283 

Spring  Subscore  Data 


Content  Organization 


R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

99 

20 

112 

231 

NSH 

31 

4 

15 

50 

S 

0 

0 

0 

0 

Total 

130 

24 

127 

281 

Language 


R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

72 

16 

80 

168 

NSH 

37 

7 

30 

74 

S 

21 

1 

17 

39 

Total 

130 

24 

127 

281 

44 


Writing  Conventions 


R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

64 

12 

74 

150 

NSH 

29 

4 

27 

60 

S 

37 

8 

26 

71 

Total 

130 

24 

127 

281 

45 


